Dual model for fisheye lens distortion and an algorithm for calibrating model parameters

ABSTRACT

An example system enabling a dual fisheye model and calibration is described. The system includes a calibration module that simultaneously generates a first set of coefficients of an inverse distortion polynomial f(ρ) representing radial distortion and a second set of coefficients of an alternative distortion polynomial g(ω). Further, the present techniques may also calibrate intrinsic and extrinsic parameters.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application claims the benefit of the filing date of U.S.Patent Provisional Application Ser. No. 62/692,937, by Radka Tezaur,entitled “Dual Model for Fisheye Lens Distortion and an Algorithm forCalibrating Model Parameters,” filed Jul. 2, 2018, and which isincorporated herein by reference.

BACKGROUND

A fisheye camera is a camera that includes a wide-angle lens thatproduces a strong visual distortion intended to create a wide panoramicor hemispherical image. Fisheye lenses may be used to capture ultra-wideangles of view.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating an example system for enabling adual fisheye model and calibrator;

FIG. 2 is a fisheye image;

FIG. 3 is an illustration of the projection of the fisheye image;

FIG. 4 is an illustration a checkerboard template;

FIG. 5A is a process flow diagram illustrating a method for a dual modelfor fisheye lens distortion and an algorithm for calibrating modelparameters via an off-line calibration;

FIG. 5B is a process flow diagram illustrating a method for a dual modelfor fisheye lens distortion and an algorithm for calibrating modelparameters via a dynamic calibration;

FIG. 6 a block diagram is shown illustrating an example computing devicethat can enable a dual fisheye model and calibrator;

FIG. 7 is a block diagram showing computer readable media that storecode for a dual fisheye model and calibrator; and

FIG. 8 is an illustration of vehicles with fisheye cameras.

The same numbers are used throughout the disclosure and the figures toreference like components and features. Numbers in the 100 series referto features originally found in FIG. 1; numbers in the 200 series referto features originally found in FIG. 2; and so on.

DESCRIPTION OF THE EMBODIMENTS

With the advance of computational imaging and computer vision, fisheyelenses are becoming more and more ubiquitous. Their use is increasinglypopular for automotive applications, security surveillance, 360 videocapture, etc., where their extremely wide field of view is very useful.In past their use was limited by the large image distortion that theycause. But, as image data processing is becoming faster and cheaper,this is obstacle is disappearing.

Computer vision and computational imaging applications require theability to map points in captured 2D images to light rays and/or pointsin 3D world and vice versa. In order to be able to do that, it isnecessary to be able to accurately calibrate the camera lens system. Themodels and algorithms used for regular cameras are, generally,unsuitable for the fisheye cameras, as they cannot handle a field ofview that can exceed 180° and the large geometric distortion associatedwith such a large field of view.

In recent years, a number of models and calibration methods have beenproposed for fisheye and other omnidirectional cameras. However, theiraccuracy and computational efficiency remain to be an issue. Currently,to characterize the radial distortion of the image, the most successfulomnidirectional camera calibration tools use a polynomial thatdetermines the coordinates of a 3D point corresponding to a 2D imagepoint.

This polynomial based model suffers some problems that have a negativeimpact on both the accuracy of calibration and the computationalefficiency (of the calibration process itself as well as of many of theapplications using the calibration results). In particular, the lensradial distortion model typically used is inefficient for mapping 3Dworld points to points in 2D captured image. Additionally, the choice ofpolynomial coefficients representing the distortion is suboptimal forcalibration accuracy. Moreover, the affine transform comprising the 2×2stretch matrix and the distortion center cannot fully compensate for thelens-sensor misalignment. Further, the stretch matrix includes aredundant parameter that can negatively impact the convergence of thecalibration process and the extrinsic calibration of multi-camerasystems.

To estimate parameters of the polynomial used to characterize a radialdistortion of the image, a calibration tool requires a sequence ofimages of a checkerboard calibration chart, such as the chart in FIG. 4.The detected chart corner points are used to estimate the parameterscharacterizing the camera. The estimated camera intrinsic parametersinclude the position of the center of the radial distortion in thecaptured images, the elements of a stretch matrix that can help tocompensate for some manufacturing imperfections, and the coefficients ofa polynomial that is used to model the lens distortion. The algorithmalso produces extrinsic parameters that characterize the position of thecamera with respect to each calibration chart position.

The present disclosure relates generally to techniques that enable adual model for fisheye lens distortion and an algorithm for calibratingmodel parameters. The dual distortion model includes both of an inversedistortion polynomial and an alternative distortion polynomial withcoefficients selected as described below. The presented fisheyecalibration techniques use the dual distortion model, which is efficientboth for mapping two-dimensional (2D) image points to thethree-dimensional (3D) world and from the 3D world to 2D image points.However, the model includes some further modifications to the underlyingimage formation model that eliminate the other mentioned disadvantages.

While a checkerboard calibration chart has been illustrated, someembodiments may not use images of checkerboard calibration chart. Forexample, other implementations may use images of a different chart or aspecial calibration device (e.g., calibration wand with LEDs), or theycould use general images that do not include any calibration chart orspecial devices. Thus, offline calibration may be performed via ageneral calibration chart or another device, such as a calibrationdevice. The dual fisheye model could be used, for example, for a dynamiccalibration implementation, which directly uses the fisheye images thatneed to be processed by some application. Thus, the present techniquesinclude off-line calibration (e.g. with the checkerboard) and a dynamiccalibration case.

FIG. 1 is a block diagram illustrating an example system for a dualmodel for fisheye lens distortion and an algorithm for calibrating modelparameters. The example system is referred to generally by the referencenumber 100 and can be implemented in the computing device 600 below inFIG. 6 using the method 500 of FIG. 5 below.

The example system 100 includes a plurality of cameras 102, a computingdevice 104, and a display 106. The computing device 104 includes areceiver 108, a dual fisheye distortion modeler 110, a calibrator 112,and a transmitter 114. The display 106 includes a display application120.

As shown in FIG. 1, a plurality of cameras 102 may capture images orvideo. Each camera may be a fisheye camera. As used herein, a fisheyecamera is a camera that includes a wide-angle lens that produces strongvisual distortion intended to create a wide panoramic or hemisphericalimage. The fisheye camera may capture an image such as the image 200 ofFIG. 2.

In embodiments, there are one or more cameras that feed the calibrationimages to the calibrator 112. However, the calibrator does not need tobe connected to a display application. The calibrator 112 produces thecalibration parameters (dual distortion model coefficients, cameraintrinsic parameters, and in the case of multiple cameras also theextrinsic parameters that characterize their mutual position) and thesecoefficients are fed to a separate system that involves cameras thathave been calibrated, a computing device that hosts the application thatprocesses the captured images, and possibly also a display. The display,however, is optional, as the fisheye images can be used, for example,for robot/autonomous vehicle navigation 122, in which case thecalibration parameters are not used to produce some rectified orcombined images to display, but they are used to find the position ofthe robot/vehicle and other objects in the scene. Thus, the calibrationparameters can be used to find the position of a robot or vehicleincluding the computing device 104. The computing device 104 may also beused to find the position of objects within the scene in real time.

In the case of dynamic calibration, when the images that are to beprocessed are also used for the calibration, the calibrator and theapplication that uses the calibration reside within the same computingdevice, to which a display may or may not be attached. Similarly, asabove, the goal may not be to produce some images that need to bedisplayed, but, e.g., to find the spatial position of objects in thescene and use it for navigation.

The computing device 104 may receive images from the cameras 102 andoutput images that are modeled according to the dual camera model andcalibration parameters described herein. The receiver 108 may receivethe plurality of images and send the plurality of images to a calibrator112. The calibrator 112 may include a dual fisheye distortion modeler110 and a parameter calculator 111. As described herein, the termscalibrator and calibration module may be used interchangeably. The dualfisheye distortion modeler 110 may approximate the fisheye distortionvia a dual fisheye distortion model that includes each of a genericTaylor polynomial and an alternative fisheye radial distortion model asdescribed below. Further, as described below the modeler 110 maygenerate the coefficients of the dual polynomials in parallel orsimultaneously. In examples, the polynomial g(ω) described below is usedwhenever a point is to be projected from the 3D space to the 2D fisheyeimage, while the inverse distortion polynomial f(ρ), similar to thatused in a conventional fisheye model, is used whenever it is necessaryto identify the incident ray and/or some point in the 3D scenecorresponding to a given point in the 2D fisheye image. Thus, theinverse distortion polynomial f(ρ) is used whenever a point is projectedfrom the 2D fisheye image to the 3D space.

The parameter calculator 111 may calculate any additional parameters tobe applied to a model of the fisheye camera. These parameters arecalculated in parallel and simultaneously with the coefficients of thedual fisheye distortion modeler polynomials. Intrinsic parameters mayinclude the parameters intrinsic to the camera itself, such asparameters related to sensor geometry and alignment. In some cases, theparameters may include any remaining intrinsic parameters that are notcalculated when the coefficients of the dual fisheye distortionpolynomials are generated. These parameters may be, for example, acenter of distortion, affine parameters, and projective parameters. Inembodiments, a six-parameter projective transform may be used toaccurately model the projection onto a tilted sensor. In some examples,a checkerboard may be used to perform intrinsic calibration. Theparameter calculator may also calculate extrinsic parameters such asrotation and translation of a chart with respect to the camera for eachcalibration image.

Thus, the calibrator 112 may be used to calibrate each of the dualfisheye distortion models simultaneously. Both of these polynomials areapproximations of the real lens distortion, by calibrating bothpolynomials the coefficients are ensured to be optimized and match thereal lens distortion rather than forcing one polynomial to match theother polynomial, which includes some approximation error. Here, theundesirable matching of the error and biasing one of these models to bemore accurate is avoided. The transmitter 114 may then transmitcalibration, and any needed projections to a display application 120 ofthe display 106. For example, the image 300 of FIG. 3 may be aprojection from the fisheye image 200 of FIG. 2. The image 300 iscomputed by the display application 120 using the images captured by thecameras. In embodiments, an image capture model is a model of the imageformation of a camera sensor, which may be represented by a projectivetransform. The dual fisheye model is a central projection with radialdistortion, which is characterized by two polynomials as describedbelow. An overall camera model may include both the dual fisheye modeland the image capture model.

In some cases, a robot or autonomous vehicle navigation system 122includes the computing device 104. In such an embodiment, the output ofthe dual fisheye distortion modeler 110 and calibrator 112 may be usedto derive the position of objects in a scene. The output of the dualfisheye distortion modeler 110 and calibrator 112 may also be used toderive the position of the robot or autonomous vehicle system within anenvironment.

The diagram of FIG. 1 is not intended to indicate that the examplesystem 100 is to include all of the components shown in FIG. 1. Rather,the example system 100 can be implemented using fewer or additionalcomponents not illustrated in FIG. 1 (e.g., additional cameras,computing devices, components, head mounted displays, etc.).

The present techniques can generate a more accurate calibration via achoice of powers in the dual fisheye distortion model. In particular, afirst polynomial of the dual fisheye distortion model may be modified touse even powers. To further improve the calibration accuracy, thefisheye distortion model used for the mapping from the ideal image planeto the actual sensor pixel array may be modified. In particular, thetypical affine transform may be replaced with a projective transform asdescribed below. Finally, in a non-linear optimization (bundleadjustment) stage the coefficients of both of these polynomials may becalibrated simultaneously.

The mapping between points in 3D space and the points in the capturedimage can be split to two phases. First, mapping from a 3D space to theideal image plane and second, mapping from the ideal image plane to theimage sensor plane. The typical model assumes that the first mapping isa central projection with radial distortion. To characterize the radialdistortion, the typical model uses a polynomial of degree four:f(ρ)=f ₀ +f ₂ρ² +f ₃ρ³ +f ₄ρ⁴

For any given point (x, y) in the ideal image plane (i.e., for any givenrefracted ray), this polynomial makes it possible to find thecorresponding incident ray by providing a point in 3D space that the raypasses through: (x, y, f(ρ)), where ρ=√{square root over (x²+y²)}. Notethat there are only four parameters in this model, as the linear term isskipped.

While this model is efficient for finding the incident ray thatcorresponds to the given image point, it is cumbersome when it comes tofinding the image point corresponding to a given point in the 3D scene.The projection of point ({tilde over (z)}, {tilde over (y)}, {tilde over(z)}) in the ideal image plane is the point (x,y) such that:s({tilde over (z)},{tilde over (y)},{tilde over (z)})=(x,y,f(ρ))for some scaling factor s>0 and ρ=√{square root over (x²+y²)}. Thismeans that:x=s“{tilde over (x)}”,y=s {tilde over (y)}and:

${s = \frac{\rho}{\overset{\sim}{\rho}}},$where {tilde over (ρ)}=√{square root over ({tilde over (x)}²+{tilde over(y)}²)}. However, to compute s, it is necessary first to find ρ. Thiscan be achieved by exploiting the relationship between the z-components:

${z = {{f(\rho)} = {{s\overset{\sim}{z}} = {\frac{\rho}{\overset{\sim}{\rho}}\overset{\sim}{z}}}}},$which yields

${f_{0} + {\frac{\overset{\sim}{z}}{\overset{\sim}{\rho}}\rho} + {f_{2}\rho^{2}} + {f_{3}\rho^{3}} + {f_{4}\rho^{4}}} = 0.$

To find the image point corresponding to the given 3D scene point, theroot of a polynomial of degree 4 is found, which is computationallyexpensive.

For the mapping between the ideal image plane (which is perpendicular tothe optical axis and is equipped with the xy-coordinate system that hasorigin at the principal point, where the optical axis intersects theplane) and the captured image (using pixel coordinates with the originin the upper left corner of the image) the typical model uses a 2Daffine transform. For the given point (x,y) in the ideal image plane thecorresponding image point is (u, v),

$\begin{pmatrix}u \\v\end{pmatrix} = {{\begin{pmatrix}s_{11} & s_{12} \\s_{21} & 1\end{pmatrix}\begin{pmatrix}x \\y\end{pmatrix}} + {\begin{pmatrix}c_{1} \\c_{2}\end{pmatrix}.}}$

Here, (c_(u), c_(v)) is the center of the radial distortion in the image(point on the optical axis). The affine transform matrix (the stretchmatrix) has only three parameters, as one of the diagonal entries canalways be normalized to 1. The coefficients of the stretch matrix aredetermined only up to scale (thus allowing the above-mentionednormalization). The scaling of the stretch matrix coefficients can beseen as either changing the scale on the x- and y-axis in the idealimage plane or changing the distance of the ideal image plane from theoptical center (the center of projection). Either way, the differentscaling of the stretch matrix corresponds to scaling the radial distanceρ in the ideal image plane and can be absorbed by adjusting thedistortion polynomial coefficients in the appropriate manner.

The calibration of the fisheye model involves two stages. First, theinitial values of the parameters are estimated using a direct method andsecond, this initial estimate is improved by iterative non-linearoptimization (bundle adjustment). The parameters to be estimated includethe intrinsic parameters listed above—f₀, f₂′ f₃, f₄, c₁, c₂, s₁₁, s₁₂,and s₂₁, and a 3D rotation matrix R_(j) (which can be parameterized bythree parameters) and translation vector t_(j) for each calibrationimage. The described algorithm for the calibration should be seen onlyas one embodiment. Other embodiments may, for example, use only directestimation of the parameters and not include the iterative non-linearoptimization step, or use some different cost function for thenon-linear optimization step. For example, the cost function may besine-based.

To obtain an initial estimate for the calibration, it is assumed thatthe center of distortion (c₁, c₂) is in the center of the image and thatthe stretch matrix is a 2×2 identity matrix. This enables finding apoint in the ideal image plane, (x, y), for each chart corner pointdetected in the captured calibration images, (u, v). The equations thatneed to be solved to find the distortion polynomial coefficients and theentries of the rotation matrices and translations vectors are derivedusing the fact that vectors (x, y, f(ρ))^(T) and R_(j) ({tilde over(x)}, {tilde over (y)}, 0)^(T)+t_(j) (representing the actual point onthe calibration chart expressed in the camera coordinate system) areparallel and thus their cross product is equal to zero.

The subsequent non-linear bundle adjustment optimizes all parameters byminimizing the total square re-projection error:

${\sum\limits_{j,k}^{\;}\left( {u_{j,k} - {\overset{\sim}{u}}_{j.k}} \right)^{2}} + \left( {v_{j,k} - {\overset{\sim}{v}}_{j.k}} \right)^{2}$

where (u_(j,k), y_(j,k)) is the j-th corner point detected in the k-thcalibration image, and (ũ_(j,k), {tilde over (v)}_(j,k)) is the actualj-th chart corner point projected to the 2D image using the estimatedintrinsic parameters and the rotation and translation for the k-thcalibration image.

In practice, the direct optimization of all the parameters by anon-linear least squares method using the initial calibration as thestarting point may not produce the desired results and some additionalsteps yield a better starting point for the final bundle adjustment toobtain sufficiently accurate calibration. For example, the typical modelincludes an additional step before the non-linear optimization stagewhich improves the estimate of the distortion center by a brute forcesearch. This step is not necessary when a slightly modified non-linearoptimization cost function is used. However, some additional steps maybe done to guarantee the convergence to a good local minimum. While thebrute force search for better estimate of the center of distortion wasunnecessary, the non-linear optimization is split into two stages,initially keeping the stretch matrix set to the identity matrix, inorder for the algorithm to converge and produce good calibration.

The set of intrinsic parameters for the typical model is not minimal.One of the parameters is redundant and can be eliminated. With all theparameters present, the same mapping between points in 3D space and inthe 2D fisheye image can be represented by infinitely many differentcombinations of the parameter values. This is because a virtual cameracoordinate system may be arbitrarily rotated about the optical axis, andcompensation for this rotation may be done by appropriately modifyingthe chart-to-camera rotations. The results of typical intrinsiccalibration may include an unnecessary small rotation between the idealimage plane and sensor coordinates of about one degree (1°). While thisis not an issue when calibrating a single camera (since this rotation isfully compensated for by the extrinsic rotation), when calibrating acamera array this may falsely indicate that the cameras are not alignedas they should be. Moreover, the redundant parameter negatively impactsthe convergence of non-linear optimization.

To eliminate the arbitrary rotation between the ideal image planecoordinates and the sensor (captured image) coordinates, the selection,for example, of s₁₂=0 (which corresponds to requiring that the image ofthe horizontal axis of the camera coordinate system in the capturedimage is horizontal on the sensor) can be made. After constraining theparameter s₁₂ to 0, the bundle adjustment of all remaining parametersdirectly produces a good calibration and it is no longer necessary tosplit the calibration into two stages, with the first stage leaving outthe stretch matrix elements from the optimization. Thus, during a directestimation of the dual polynomial coefficients, the image of thehorizontal axis of the camera coordinate system in the captured image ishorizontal and stretch matrix elements are not used.

The fisheye lens design, in general, attempts to follow some commondistortion model. This model may be an equidistant model, in which theimage point height is proportional to the incident angle, ρ=fω, or astereographic model, in which ρ=2f tan(ω/2). However, the actual lensdesign inevitably does not match the model perfectly and using a modelwith a single parameter f (lens focal length) does not produce anaccurate enough lens calibration. The function of the incident angle inthese distortion models may be replaced by a Taylor expansion, and thecoefficients of the Taylor polynomial are calibrated. Since thefunctions representing the distortion are always odd, only the odddegree coefficients of the Taylor expansion are non-zero and arecalibrated as follows:

$\rho = {{g(\omega)} = {\sum\limits_{n = 0}^{N}{g_{n}{\omega^{{2n} + 1}.}}}}$

The advantage of this second, alternative fisheye radial distortionmodel is that the projection of the 3D scene point to the 2D fisheyeimage can be computed efficiently. Unlike the typical distortion model,finding an image point height p only requires evaluating the polynomial,not finding its roots. However, this model makes it more complicated tomap the points in the opposite direction, from the 2D fisheye image tothe incident rays and/or points in the 3D scene, as solving for theincident angle requires finding the root of a polynomial, typically ofdegree 5 or higher.

To improve the computational efficiency, both of the calibration processand of the applications using the resulting camera calibration, thefirst and second distortion models are combined to find a dual fisheyedistortion model. The dual fisheye model uses the first and seconddistortion models simultaneously. The calibration process finds thecoefficients of each of these polynomials in parallel. The polynomialg(ω) described above is used whenever a point needs to be projected fromthe 3D space to the 2D fisheye image, while the inverse distortionpolynomial f(ρ), similar to a typical fisheye model, is used whenever itis necessary to identify the incident ray and/or some point in the 3Dscene corresponding to a given point in the 2D fisheye image.

The polynomial f(ρ) as described according to the present techniquesdiffers from that used in typical fisheye modeling. While the typicalfisheye models use the following:f(ρ)=f ₀ +f ₂ρ² +f ₃ρ³ +f ₄ρ⁴,the present techniques use even powers:

${f(\rho)} = {\sum\limits_{m = 0}^{M}{f_{m}{\rho^{2m}.}}}$

In embodiments, f can be viewed as the function of two variables, x andy, that is radially symmetric. Requiring the linear term coefficient f₁to be zero in the typical model prevents the first derivative of f frombeing discontinuous at zero. This argument can be extended to higherorder derivatives. In order to make also all the higher orderderivatives continuous, f(ρ) should be an even function, where all odddegree coefficients of f(ρ) are zero. Similarly, as in the typicalmodel, f(ρ) may be used with four parameters that are optimized.According to the present techniques, it is assumed that:f(P)=f ₀ +f ₁ρ² +f ₂ρ⁴ +f ₃ρ⁶.

This choice of powers yields a more accurate calibration than the oneused by typical models. To further improve the calibration accuracy, themodel used for the mapping from the ideal image plane to the actualsensor pixel array may also be modified. In practice, due to themanufacturing tolerances, the optical elements and the sensor may not beperfectly aligned. The affine model used by typical models cancompensate for the sensor being shifted with respect to the lens, i.e.,for not having the center of distortion exactly in the center of theimage. However, the affine model of the typical model is not able tofully compensate for the sensor not being perfectly perpendicular to theoptical axis. Thus, the affine transform in of the typical model isreplaced with a projective transform, as it can accurately model theprojection onto a tilted sensor.

While the projective transform has, in general, eight degrees offreedom, using eight parameters would lead to a redundantparameterization. It can be shown that only six parameters are neededand the homography matrix representing the projective transform from theideal image plane to the sensor (or the captured image) can be expressedas:

$H = {\begin{pmatrix}1 & 0 & c_{1} \\0 & 1 & c_{2} \\0 & 0 & 1\end{pmatrix}\begin{pmatrix}1 & 0 & 0 \\0 & 1 & 0 \\p_{1} & p_{2} & 1\end{pmatrix}\begin{pmatrix}a_{1} & a_{2} & 0 \\0 & 1 & 0 \\0 & 0 & 1\end{pmatrix}}$

Here, (c₁,c₂) are the coordinates of the distortion center in thecaptured images, p₁ and p₂ are new projective parameters, and a₁ and a₂are affine parameters. Factoring the homography matrix in this manner isadvantageous for several reasons. Most importantly, it enables efficientimplementation of both the projective transform from the ideal imageplane to the sensor pixel array and its inverse, since the homographymatrix representing the inverse transform is:

$H^{- 1} = {\begin{pmatrix}{1/a_{1}} & {{- a_{2}}/a_{1}} & 0 \\0 & 1 & 0 \\0 & 0 & 1\end{pmatrix}\begin{pmatrix}1 & 0 & 0 \\0 & 1 & 0 \\{- p_{1}} & {- p_{2}} & 1\end{pmatrix}\begin{pmatrix}1 & 0 & {- c_{1}} \\0 & 1 & {- c_{2}} \\0 & 0 & 1\end{pmatrix}}$

Another advantage is that this representation makes it possible toeasily switch between the projective model and the affine model. Whilethe affine model is less accurate, sometimes it can be preferred inpractice, because unlike the projective transform, affine transform doesnot require divisions which can be expensive to implement. To produce acalibration that uses an affine transform only, the projectiveparameters p₁ and p₂ are set to zero. The affine parameters a₁ and a₂are equivalent to the typical model stretch matrix elements s₁₁ and s₁₂.

While it would be possible to use the typical algorithm to calibrate oneof the dual polynomials and then to match the other polynomial to thecalibrated polynomial by performing a least-squares fit, such anapproach is prone to an undesirable matching of the error in thecalibrated polynomial as well as biasing of the polynomial. Inembodiments, a first polynomial may be matched to a second polynomial iscalculate an initial estimate, but in the non-linear optimization(bundle adjustment) stage the coefficients of both polynomials arecalibrated simultaneously. In this case, both polynomials areapproximations of the real lens distortion, and calibrating both ensuresthat the coefficients are optimized to match the real lens distortionrather than forcing one of them only to match the other polynomial,which includes some approximation error. In this manner, the undesirablematching of the error and biasing one of these models to be moreaccurate is avoided.

One embodiment of the cost function that is used for bundle adjustmenthas three parts: First, a reprojection error of projecting the chartcorner points from 3D to the fisheye images. Second, an angle error ofthe projected incident rays corresponding to the chart corner pointsdetected in the 2D fisheye images. Finally, a reprojection error ofprojecting the detected chart corner points to incident rays in 3D andfrom there back to 2D fisheye image.

The following cost function is minimized:

${\alpha_{1}{\sum\limits_{j,k}^{\;}{{u_{j,k} - {\overset{\sim}{u}}_{j.k}}}^{2}}} + {\alpha_{2}{\sum\limits_{j,k}^{\;}{{a\cos}^{2}\left( {x_{j,k}^{T}{\overset{\sim}{x}}_{jk}} \right)}}} + {\alpha_{3}{\sum\limits_{j,k}^{\;}{{u_{j,k} - {\hat{u}}_{j.k}}}^{2}}}$Alternatively, the cost function may use the sine of the angle (insteadof the angle between rays):

${\alpha_{1}{\sum\limits_{j,k}^{\;}{{u_{j,k} - {\overset{\sim}{u}}_{j.k}}}^{2}}} + {\alpha_{2}{\sum\limits_{j,k}^{\;}\left( {1 - \left( {x_{j,k}^{T}{\overset{\sim}{x}}_{jk}} \right)^{2}} \right)}} + {\alpha_{3}{\sum\limits_{j,k}^{\;}{{u_{j,k} - {\hat{u}}_{j.k}}}^{2}}}$

Here, variable u_(j,k) is the j-th chart corner point detected in thek-th calibration image. Variable x_(j,k) is the point on incident raycorresponding to this point (computed with the current estimate of thecamera intrinsic parameters), normalized to lie on the unit sphere,∥x_(j,k)∥=1. Variable û_(j,k) is x_(j,k) projected back to the fisheyeimage (using the current estimate of the camera intrinsic parameters).Variable {tilde over (x)}_(jk) is the j-th corner point of thecalibration chart placed in the position depicted in the k-thcalibration image, expressed in the camera coordinate system (using thecurrent estimate of the camera to chart extrinsic parameters),normalized to lie on the unit sphere, ∥{tilde over (x)}_(jk)∥=1.Variable ũ_(j,k) is {tilde over (x)}_(jk) projected to the fisheye image(using the current estimate of the camera intrinsic parameters).Finally, variables α₁, α₂, and α₃ are the weights that can be tuned tobalance the contributions of the three parts of the cost function.

Each of the three components of the cost function has its own purpose.Computing the first component involves projecting points from 3D to thefisheye images, which employs the values of the coefficients of thedistortion polynomial g(ω). This term is used to optimize the parametersg_(m), m=0, . . . , M. Evaluating the second term involves findingpoints x_(j,k) that determine the incident rays in 3D which correspondto the detected corner points in the fisheye images. This requires usingthe inverse distortion polynomial f(ρ) and thus it is used to optimizeits parameters, f_(n), n=0, . . . , N. Finally, evaluating the last termensures that at the parameters g_(m), m=0, . . . , M and the parametersf_(n), n=0, . . . , N, for which the performance of polynomials g(ω) andf(ρ) is being optimized, match such that the polynomials represent thesame fisheye distortion.

To improve the robustness with respect to incorrectly detected chartcorner points, the cost function can be used with errors weightedaccording to the estimated reliability of each point as follows:

${\alpha_{1}{\sum\limits_{j,k}^{\;}{w_{j,k}^{2}{{u_{j,k} - {\overset{\sim}{u}}_{j.k}}}^{2}}}} + {\alpha_{2}{\sum\limits_{j,k}^{\;}{w_{j,k}^{2}{{a\cos}^{2}\left( {x_{j,k}^{T}{\overset{\sim}{x}}_{jk}} \right)}}}} + {\alpha_{3}{\sum\limits_{j,k}^{\;}{w_{j,k}^{2}{{u_{j,k} - {\hat{u}}_{j.k}}}^{2}}}}$Or, alternatively,

${\alpha_{1}{\sum\limits_{j,k}^{\;}{w_{j,k}^{2}{{u_{j,k} - {\overset{\sim}{u}}_{j.k}}}^{2}}}} + {\alpha_{2}{\sum\limits_{j,k}^{\;}{w_{j,k}^{2}\left( {1 - \left( {x_{j,k}^{T}{\overset{\sim}{x}}_{jk}} \right)^{2}} \right)}}} + {\alpha_{3}{\sum\limits_{j,k}^{\;}{w_{j,k}^{2}{{u_{j,k} - {\hat{u}}_{j.k}}}^{2}}}}$

The weights q_(j,k) can be set, for example, asw _(j,k)=min(1,∥u _(j,k) −ũ _(j,k)∥⁻²)

where ũ_(j,k) are reprojected chart corner points obtained with someprevious values of the intrinsic and extrinsic parameters.

To get a close enough estimate of the parameter values suitable forinitializing the bundle adjustment, an algorithm similar to typicalmodel is used. However, to avoid the need for using a constrainedleast-squares fit when finding the inverse distortion polynomialcoefficients, a step is added to the process. First, the coefficients ofthe inverse distortion polynomial f(ρ) are estimated up to a scalefactor, using the fact that the computed incident ray points x_(j,k)should be co-planar for all the chart corner points that lie on a line.Then, in the last step of the process, in which the typical modelestimates these coefficients, the missing scale factor is estimated.

FIG. 5A is a process flow diagram illustrating a method 500A for a dualmodel for fisheye lens distortion and an algorithm for calibrating modelparameters via an off-line calibration. The example method is generallyreferred to by the reference number 500A and can be implemented usingthe computing device 104 of FIG. 1 above, the processor 602 of thecomputing device 600 of FIG. 6 below, the computer readable media 700 ofFIG. 7 below, or the vehicle 800 of FIG. 8 below.

At block 502, the coefficients of an inverse distortion polynomial f(ρ),the coefficients of an alternative fisheye polynomial g(ω), and all theremaining intrinsic and extrinsic parameters are estimated by via directestimation. Here, calibrating the dual polynomials includessimultaneously generating coefficients for the inverse distortionpolynomial and the alternative polynomial. Moreover, other intrinsicparameters may be calculated in parallel with the coefficients for theinverse distortion polynomial and the alternative polynomial.

In some cases, an initial estimate for the calibration of the inversedistortion polynomial f(ρ) and the alternative fisheye polynomial g(ω)is calculated prior to calibrating the dual polynomials at block 502.The initial estimate for the calibration parameters may include dualdistortion model coefficients, and the extrinsic parameters thatcharacterize the mutual position of a camera with respect to thecalibration chart. In particular, first the coefficients of the inversedistortion polynomial f(ρ) are estimated up to scale, then the extrinsicparameters are calculated with the scale for the coefficients of theinverse distortion polynomial. Finally, the coefficients of thealternative fisheye polynomial g(ω) coefficients are estimated. Theother intrinsic parameters are estimated in the case of an initialestimate, and instead default values representing an ideal alignment areused. Default values include the center of distortion being in thecenter of the image. In an off-line calibration scenario, a particularimage is captured and used to find the initial estimate of thecalibration parameters. For example, a checkerboard image with knowndimensions may be used to obtain the initial estimate of the calibrationparameters.

At block 504, iterative non-linear optimization of all parameters isapplied simultaneously. Thus, the calibration coefficients andparameters found simultaneous calibration of the dual fisheyepolynomials may be iteratively optimized. In particular, the initialestimate is improved by bundle adjustment. The bundle adjustment may bea three-part bundle adjustment as described above.

FIG. 5B is a process flow diagram illustrating a method 500B for a dualmodel for fisheye lens distortion and an algorithm for calibrating modelparameters via a dynamic calibration. The example method is generallyreferred to by the reference number 500B and can be implemented usingthe computing device 104 of FIG. 1 above, the processor 602 of thecomputing device 600 of FIG. 6 below, the computer readable media 700 ofFIG. 7 below, or the vehicle 800 of FIG. 8 below.

At block 512, the coefficients of an inverse distortion polynomial f(ρ),an alternative fisheye model g(ω), and all the remaining intrinsic andextrinsic parameters are initialized. They may represent, for example,factory calibration stored in the memory.

At block 514, dynamic calibration of coefficients of the inversedistortion polynomial, coefficients of the alternative fisheyepolynomial, intrinsic parameters, and/or extrinsic parameters isperformed by using key points detected in the images. Dynamiccalibration is the calibration using images captured by at least onefisheye camera that are to be used by the display application or formapping or navigation functionality. Dynamic calibration is used whenthe properties of the imaging system change significantly over time.

In dynamic calibration, a plurality of matching point pairs is detectedin calibration images. The 3D coordinates of the corresponding pointsmay be determined along with depth values for the detected featurepoints. Typically, dynamic calibration uses RANSAC method to providerobustness with respect to incorrectly matched points. In embodiments,the dynamic calibration may be updated periodically, or as necessarybased on any image captured by the fisheye camera.

In FIG. 5A, any number of additional blocks not shown may be includedwithin the example process 500A, depending on the details of thespecific implementation. Similarly, in FIG. 5B, any number of additionalblocks not shown may be included within the example process 500B,depending on the details of the specific implementation.

In this manner, methods 500A and 500B of FIGS. 5A and 5B enable the dualfisheye distortion model to use the alternative modified polynomial g(ω)whenever a point needs to be projected from the 3D space to the 2Dfisheye image, while the modified inverse distortion polynomial f(ρ) isused whenever it is necessary to identify the incident ray and/or somepoint in the 3D scene corresponding to a given point in the 2D fisheyeimage.

Referring now to FIG. 6, a block diagram is shown illustrating anexample computing device that enables a dual model for fisheye lensdistortion and an algorithm for calibrating model parameters. Thecomputing device 600 may be, for example, a laptop computer, desktopcomputer, tablet computer, mobile device, or wearable device, amongothers. In some examples, the computing device 600 may be a smart cameraor a digital security surveillance camera. The computing device 600 mayinclude a central processing unit (CPU) 602 that is configured toexecute stored instructions, as well as a memory device 604 that storesinstructions that are executable by the CPU 602. The CPU 602 may becoupled to the memory device 604 by a bus 606. Additionally, the CPU 602can be a single core processor, a multi-core processor, a computingcluster, or any number of other configurations. Furthermore, thecomputing device 600 may include more than one CPU 602. In someexamples, the CPU 602 may be a system-on-chip (SoC) with a multi-coreprocessor architecture. In some examples, the CPU 602 can be aspecialized digital signal processor (DSP) used for image processing.The memory device 604 can include random access memory (RAM), read onlymemory (ROM), flash memory, or any other suitable memory systems. Forexample, the memory device 604 may include dynamic random-access memory(DRAM).

The memory device 604 can include random access memory (RAM), read onlymemory (ROM), flash memory, or any other suitable memory systems. Forexample, the memory device 604 may include dynamic random-access memory(DRAM). The memory device 604 may include device drivers 610 that areconfigured to execute the instructions for a dual model for fisheye lensdistortion and an algorithm for calibrating model parameters. The devicedrivers 610 may be software, an application program, application code,or the like.

The computing device 600 may also include a graphics processing unit(GPU) 608. As shown, the CPU 602 may be coupled through the bus 606 tothe GPU 608. The GPU 608 may be configured to perform any number ofgraphics operations within the computing device 600. For example, theGPU 608 may be configured to render or manipulate graphics images,graphics frames, videos, or the like, to be displayed to a user of thecomputing device 600.

The memory device 604 can include random access memory (RAM), read onlymemory (ROM), flash memory, or any other suitable memory systems. Forexample, the memory device 604 may include dynamic random-access memory(DRAM). The memory device 604 may include device drivers 610 that areconfigured to execute the instructions for a dual model for fisheye lensdistortion and an algorithm for calibrating model parameters. The devicedrivers 610 may be software, an application program, application code,or the like.

The CPU 602 may also be connected through the bus 606 to an input/output(I/O) device interface 612 configured to connect the computing device600 to one or more I/O devices 614. The I/O devices 614 may include, forexample, a keyboard and a pointing device, wherein the pointing devicemay include a touchpad or a touchscreen, among others. The I/O devices614 may be built-in components of the computing device 600 or may bedevices that are externally connected to the computing device 600. Insome examples, the memory 604 may be communicatively coupled to I/Odevices 614 through direct memory access (DMA).

The CPU 602 may also be linked through the bus 606 to a displayinterface 616 configured to connect the computing device 600 to adisplay device 618. The display devices 618 may include a display screenthat is a built-in component of the computing device 600. The displaydevices 618 may also include a computer monitor, television, orprojector, among others, that is internal to or externally connected tothe computing device 600. The display device 618 may also include a headmounted display. For example, the head mounted display may receivestereoscopic light field view corresponding to a particular perspective.For example, the head mounted display can detect a translation and sendupdated coordinates corresponding to the perspective to the receiver 632described below.

The computing device 600 also includes a storage device 620. The storagedevice 620 is a physical memory such as a hard drive, an optical drive,a thumbdrive, an array of drives, a solid-state drive, or anycombinations thereof. The storage device 620 may also include remotestorage drives.

The computing device 600 may also include a network interface controller(NIC) 622. The NIC 622 may be configured to connect the computing device600 through the bus 606 to a network 624. The network 624 may be a widearea network (WAN), local area network (LAN), or the Internet, amongothers. In some examples, the device may communicate with other devicesthrough a wireless technology. For example, the device may communicatewith other devices via a wireless local area network connection. In someexamples, the device may connect and communicate with other devices viaBluetooth® or similar technology.

The computing device 600 further includes a camera interface 626. Forexample, the camera interface 626 may be connected to a plurality ofcameras 628. In some examples, the cameras 628 may include wide anglelenses, ultra-wide angle lenses, fisheye lenses, or any combinationthereof.

The computing device 600 further includes a dual fisheye model andcalibrator 630. For example, the dual fisheye model and calibrator 630can be used for dual fisheye lens modeling and calibration. The dualfisheye model and calibrator 630 can include a receiver 632,coefficient/parameter generator 634, an iterative optimizer/directestimator 636, and a transmitter 638. In some examples, each of thecomponents 632-638 of the dual fisheye model and calibrator 630 may be amicrocontroller, embedded processor, or software module. A receiver 632may obtain camera specific values. A coefficient/parameter generator 634may determine the coefficients of an inverse distortion polynomial f(ρ)to model radial distortion. The coefficient/parameter generator 634 mayalso determine the coefficients of an alternative fisheye model g(ω).Finally, the coefficient/parameter generator 634 may also determineintrinsic parameters. The coefficients of the inverse distortionpolynomial, the coefficients of the alternative fisheye model, and atleast a portion of the intrinsic parameters are calculatedsimultaneously, or in parallel.

An iterative optimizer 636 may perform iterative non-linear optimizationof all parameters of f(ρ) and g(ω) simultaneously. A transmitter 638 maytransmit the dual fisheye distortion model and calibration values asneeded. The dual fisheye distortion model can use the alternativemodified polynomial g(ω) whenever a point needs to be projected from the3D space to the 2D fisheye image, while the modified inverse distortionpolynomial f(ρ) is used whenever it is necessary to identify theincident ray and/or some point in the 3D scene corresponding to a givenpoint in the 2D fisheye image.

The block diagram of FIG. 6 is not intended to indicate that thecomputing device 600 is to include all of the components shown in FIG.6. Rather, the computing device 600 can include fewer or additionalcomponents not illustrated in FIG. 6, such as additional buffers,additional processors, and the like. The computing device 600 mayinclude any number of additional components not shown in FIG. 6,depending on the details of the specific implementation. Furthermore,any of the functionalities of the receiver 632, coefficient/parametergenerator 634, iterative optimizer 636, transmitter 638, a calibrator,or a calibration module may be partially, or entirely, implemented inhardware and/or in the processor 602. For example, the functionality maybe implemented with an application specific integrated circuit, in logicimplemented in the processor 602, or in any other device. For example,the functionality of the dual fisheye model and calibrator 630 may beimplemented with an application specific integrated circuit, in logicimplemented in a processor, in logic implemented in a specializedgraphics processing unit such as the GPU 608, or in any other device.

FIG. 7 is a block diagram showing computer readable media 700 that storecode for a dual fisheye model and calibrator. The computer readablemedia 700 may be accessed by a processor 702 over a computer bus 704.Furthermore, the computer readable medium 700 may include codeconfigured to direct the processor 702 to perform the methods describedherein. In some embodiments, the computer readable media 700 may benon-transitory computer readable media. In some examples, the computerreadable media 700 may be storage media.

The various software components discussed herein may be stored on one ormore computer readable media 700, as indicated in FIG. 7. For example, areceiver module 706 may be configured to receive camera specific values.A coefficient/parameter generation module 708 may determine thecoefficients of an inverse distortion polynomial f(ρ) to model radialdistortion, the coefficients of an alternative fisheye model g(ω), andintrinsic parameters. The coefficients of the inverse distortionpolynomial, the coefficients of the alternative fisheye model, and atleast a portion of the intrinsic parameters are calculatedsimultaneously, or in parallel. An iterative optimizer/direct estimationmodule 710 may perform iterative non-linear optimization of allparameters of f(ρ) and g(ω) simultaneously. A transmitter module 712 maytransmit the dual fisheye distortion model and calibration values asneeded.

The block diagram of FIG. 7 is not intended to indicate that thecomputer readable media 700 is to include all of the components shown inFIG. 7. Further, the computer readable media 700 may include any numberof additional components not shown in FIG. 7, depending on the detailsof the specific implementation.

FIG. 8 is an illustration of vehicles 800A and 800B with fisheyecameras. Vehicle 800A includes two fisheye cameras 802 and 804. Vehicle800B includes two fisheye cameras 806, 808, 810, and 812. While aparticular number of cameras are illustrated, vehicles 800A and 800B mayinclude any number of fisheye cameras, at any position along thesurface, within, or underneath the vehicles 800A and 800B. Further,vehicles 800A and 800B may include system 100 for a dual model forfisheye lens distortion or the computing device 104 as described byFIG. 1. Further, the vehicles 800A and 800B are able to implement themethods 500A and 500B of FIGS. 5A and 5B, respectively.

As illustrated, vehicle 800A may capture a scene from the environmentincluding various objects. Each of the fisheye cameras 802 and 804 cancapture up to a one-hundred and eighty-degree field of view (FOV).Similarly, vehicle 800B may capture a scene from the environmentincluding various objects. Each of the fisheye cameras 806, 808, 810,and 812 can capture up to a one hundred and eighty-degree FOV. Here, thevarious FOVs overlap. The present techniques enable the cameras 802,804, 806, 808, 810, and 812 map points in captured 2D images to lightrays and/or points in 3D world and vice versa via efficient cameracalibration. The calibration enables the vehicles to quickly determinetheir location within an environment.

Example 1 is a system for enabling a dual fisheye model and calibration.The system includes a calibration module that is at least partiallyhardware, to simultaneously generate a first set of coefficients of aninverse distortion polynomial representing radial distortion and asecond set of coefficients of an alternative distortion polynomial.

Example 2 includes the system of example 1, including or excludingoptional features. In this example, the first set of coefficients andthe second set of coefficients are optimized via an iterative non-linearoptimization.

Example 3 includes the system of any one of examples 1 to 2, includingor excluding optional features. In this example, the first set ofcoefficients and the second set of coefficients are generated via adirect estimation.

Example 4 includes the system of any one of examples 1 to 3, includingor excluding optional features. In this example, the calibration moduleenables offline calibration with a calibration chart or another device.

Example 5 includes the system of any one of examples 1 to 4, includingor excluding optional features. In this example, the calibration moduleenables a dynamic calibration via an image captured by a camera of acamera array.

Example 6 includes the system of any one of examples 1 to 5, includingor excluding optional features. In this example, system of claim 1,where the inverse distortion polynomial includes even powers, andexcludes odd powers.

Example 7 includes the system of any one of examples 1 to 6, includingor excluding optional features. In this example, the system includes animage capture model, wherein the image capture model comprises aprojective transform from an ideal image plane to a two-dimensional (2D)image.

Example 8 includes the system of any one of examples 1 to 7, includingor excluding optional features. In this example, an initial estimate ofthe first set of coefficients and the second set of coefficients iscalculated, and the inverse distortion polynomial and the alternativedistortion model are calibrated simultaneously using the initialestimate.

Example 9 includes the system of any one of examples 1 to 8, includingor excluding optional features. In this example, a bundle adjustment isused to simultaneously optimize the first set of coefficients and thesecond set of coefficients.

Example 10 includes the system of any one of examples 1 to 9, includingor excluding optional features. In this example, a bundle adjustmentcomprises minimizing a cost function to iteratively optimize the firstset of coefficients and the second set of coefficients.

Example 11 is a method for enabling a dual fisheye model and calibrator.The method includes calibrating coefficients of an inverse distortionpolynomial, coefficients of an alternative distortion polynomial, andcamera parameters simultaneously to model a real-world distortion of atleast one fisheye camera.

Example 12 includes the method of example 11, including or excludingoptional features. In this example, the camera parameters comprisesensor geometry parameters and alignment parameters.

Example 13 includes the method of any one of examples 11 to 12,including or excluding optional features. In this example, thecoefficients of the inverse distortion polynomial, coefficients of thealternative distortion polynomial, and the camera parameters aregenerated via iterative non-linear optimization.

Example 14 includes the method of any one of examples 11 to 13,including or excluding optional features. In this example, thecoefficients of the inverse distortion polynomial, coefficients of thealternative distortion polynomial, and the camera parameters aregenerated via a direct estimation.

Example 15 includes the method of any one of examples 11 to 14,including or excluding optional features. In this example,simultaneously the coefficients of the inverse distortion polynomial,coefficients of the alternative distortion polynomial, and the cameraparameters to model a real-world distortion of the at least one fisheyelens is performed via an offline calibration with a calibration chart oranother device.

Example 16 includes the method of any one of examples 11 to 15,including or excluding optional features. In this example,simultaneously calibrating the coefficients of the inverse distortionpolynomial, coefficients of the alternative distortion polynomial, andthe camera parameters to model a real-world distortion of the at leastone fisheye camera is performed via a dynamic calibration.

Example 17 includes the method of any one of examples 11 to 16,including or excluding optional features. In this example, the inversedistortion polynomial includes even powers, and excludes odd powers.

Example 18 includes the method of any one of examples 11 to 17,including or excluding optional features. In this example, an imagecapture model comprises projective transform from an ideal image planeto a 2D image.

Example 19 includes the method of any one of examples 11 to 18,including or excluding optional features. In this example, a bundleadjustment is used to simultaneously optimize the coefficients of theinverse distortion polynomial and the coefficients of the alternativedistortion polynomial.

Example 20 includes the method of any one of examples 11 to 19,including or excluding optional features. In this example, bundleadjustment comprising minimizing a cost function is used tosimultaneously optimize the coefficients of the inverse distortionpolynomial and the coefficients of the alternative distortionpolynomial.

Example 21 is an apparatus. The apparatus includes a camera arraycomprising at least one fisheye camera; a calibration module thatsimultaneously generates a first set of coefficients of an inversedistortion polynomial representing a radial distortion of the fisheyecamera and a second set of coefficients of an alternative distortionpolynomial representing the fisheye camera.

Example 22 includes the apparatus of example 21, including or excludingoptional features. In this example, the first set of coefficients andthe second set of coefficients are optimized via an iterative non-linearoptimization.

Example 23 includes the apparatus of any one of examples 21 to 22,including or excluding optional features. In this example, the first setof coefficients and the second set of coefficients are generated via adirect estimation.

Example 24 includes the apparatus of any one of examples 21 to 23,including or excluding optional features. In this example, thecalibration module enables offline calibration of the camera array witha calibration chart or another device.

Example 25 includes the apparatus of any one of examples 21 to 24,including or excluding optional features. In this example, thecalibration module enables a dynamic calibration via an image capturedby the fisheye camera the camera array.

Example 26 includes the apparatus of any one of examples 21 to 25,including or excluding optional features. In this example, apparatus ofclaim 21, where the inverse distortion polynomial includes even powers,and excludes odd powers.

Example 27 includes the apparatus of any one of examples 21 to 26,including or excluding optional features. In this example, the apparatusincludes an image capture model, wherein the image capture modelcomprises a projective transform from an ideal image plane to atwo-dimensional (2D) image.

Example 28 includes the apparatus of any one of examples 21 to 27,including or excluding optional features. In this example, an initialestimate of the first set of coefficients and the second set ofcoefficients is calculated, and the inverse distortion polynomial andthe alternative distortion model are calibrated simultaneously using theinitial estimate.

Example 29 includes the apparatus of any one of examples 21 to 28,including or excluding optional features. In this example, a bundleadjustment is used to simultaneously optimize the first set ofcoefficients and the second set of coefficients.

Example 30 includes the apparatus of any one of examples 21 to 29,including or excluding optional features. In this example, a bundleadjustment comprises minimizing a cost function to iteratively optimizethe first set of coefficients and the second set of coefficients.

Example 31 is at least one non-transitory machine-readable medium havinginstructions stored therein that. The computer-readable medium includesinstructions that direct the processor to calibrate coefficients of aninverse distortion polynomial, coefficients of an alternative distortionpolynomial, and camera parameters simultaneously to model a real-worlddistortion of at least one fisheye lens.

Example 32 includes the computer-readable medium of example 31,including or excluding optional features. In this example, the cameraparameters comprise sensor geometry parameters and alignment parameters.

Example 33 includes the computer-readable medium of any one of examples31 to 32, including or excluding optional features. In this example, thecoefficients of the inverse distortion polynomial, coefficients of thealternative distortion polynomial, and the camera parameters aregenerated via iterative non-linear optimization.

Example 34 includes the computer-readable medium of any one of examples31 to 33, including or excluding optional features. In this example, thecoefficients of the inverse distortion polynomial, coefficients of thealternative distortion polynomial, and the camera parameters aregenerated via a direct estimation.

Example 35 includes the computer-readable medium of any one of examples31 to 34, including or excluding optional features. In this example,simultaneously the coefficients of the inverse distortion polynomial,coefficients of the alternative distortion polynomial, and the cameraparameters to model a real-world distortion of the at least one fisheyelens is performed via an offline calibration with a calibration chart oranother device.

Example 36 includes the computer-readable medium of any one of examples31 to 35, including or excluding optional features. In this example,simultaneously calibrating the coefficients of the inverse distortionpolynomial, coefficients of the alternative distortion polynomial, andthe camera parameters to model a real-world distortion of the at leastone fisheye lens is performed via a dynamic calibration.

Example 37 includes the computer-readable medium of any one of examples31 to 36, including or excluding optional features. In this example, theinverse distortion polynomial includes even powers, and excludes oddpowers.

Example 38 includes the computer-readable medium of any one of examples31 to 37, including or excluding optional features. In this example, animage capture model comprises projective transform from an ideal imageplane to a 2D image.

Example 39 includes the computer-readable medium of any one of examples31 to 38, including or excluding optional features. In this example, abundle adjustment is used to simultaneously optimize the coefficients ofthe inverse distortion polynomial and the coefficients of thealternative distortion polynomial.

Example 40 includes the computer-readable medium of any one of examples31 to 39, including or excluding optional features. In this example,bundle adjustment comprising minimizing a cost function is used tosimultaneously optimize the coefficients of the inverse distortionpolynomial and the coefficients of the alternative distortionpolynomial.

Example 41 is an apparatus. The apparatus includes instructions thatdirect the processor to a camera array comprising at least one fisheyecamera; a means to simultaneously generate a first set of coefficientsof an inverse distortion polynomial representing a radial distortion ofthe fisheye camera and a second set of coefficients of an alternativedistortion polynomial representing the fisheye camera.

Example 42 includes the apparatus of example 41, including or excludingoptional features. In this example, the first set of coefficients andthe second set of coefficients are optimized via an iterative non-linearoptimization.

Example 43 includes the apparatus of any one of examples 41 to 42,including or excluding optional features. In this example, the first setof coefficients and the second set of coefficients are generated via adirect estimation.

Example 44 includes the apparatus of any one of examples 41 to 43,including or excluding optional features. In this example, the means tosimultaneously generate the first set of coefficients and the second setof coefficients enables offline calibration of the camera array with acalibration chart or another device.

Example 45 includes the apparatus of any one of examples 41 to 44,including or excluding optional features. In this example, the means tosimultaneously generate the first set of coefficients and the second setof coefficients enables a dynamic calibration via an image captured bythe fisheye camera the camera array.

Example 46 includes the apparatus of any one of examples 41 to 45,including or excluding optional features. In this example, the inversedistortion polynomial includes even powers, and excludes odd powers.

Example 47 includes the apparatus of any one of examples 41 to 46,including or excluding optional features. In this example, the apparatusincludes an image capture model, wherein the image capture modelcomprises a projective transform from an ideal image plane to atwo-dimensional (2D) image.

Example 48 includes the apparatus of any one of examples 41 to 47,including or excluding optional features. In this example, an initialestimate of the first set of coefficients and the second set ofcoefficients is calculated, and the inverse distortion polynomial andthe alternative distortion model are calibrated simultaneously using theinitial estimate.

Example 49 includes the apparatus of any one of examples 41 to 48,including or excluding optional features. In this example, a bundleadjustment is used to simultaneously optimize the first set ofcoefficients and the second set of coefficients.

Example 50 includes the apparatus of any one of examples 41 to 49,including or excluding optional features. In this example, a bundleadjustment comprises minimizing a cost function to iteratively optimizethe first set of coefficients and the second set of coefficients.

Not all components, features, structures, characteristics, etc.described and illustrated herein need be included in a particular aspector aspects. If the specification states a component, feature, structure,or characteristic “may”, “might”, “can” or “could” be included, forexample, that particular component, feature, structure, orcharacteristic is not required to be included. If the specification orclaim refers to “a” or “an” element, that does not mean there is onlyone of the element. If the specification or claims refer to “anadditional” element, that does not preclude there being more than one ofthe additional element.

It is to be noted that, although some aspects have been described inreference to particular implementations, other implementations arepossible according to some aspects. Additionally, the arrangement and/ororder of circuit elements or other features illustrated in the drawingsand/or described herein need not be arranged in the particular wayillustrated and described. Many other arrangements are possibleaccording to some aspects.

In each system shown in a figure, the elements in some cases may eachhave a same reference number or a different reference number to suggestthat the elements represented could be different and/or similar.However, an element may be flexible enough to have differentimplementations and work with some or all of the systems shown ordescribed herein. The various elements shown in the figures may be thesame or different. Which one is referred to as a first element and whichis called a second element is arbitrary.

It is to be understood that specifics in the aforementioned examples maybe used anywhere in one or more aspects. For instance, all optionalfeatures of the computing device described above may also be implementedwith respect to either of the methods or the computer-readable mediumdescribed herein. Furthermore, although flow diagrams and/or statediagrams may have been used herein to describe aspects, the techniquesare not limited to those diagrams or to corresponding descriptionsherein. For example, flow need not move through each illustrated box orstate or in exactly the same order as illustrated and described herein.

The present techniques are not restricted to the particular detailslisted herein. Indeed, those skilled in the art having the benefit ofthis disclosure will appreciate that many other variations from theforegoing description and drawings may be made within the scope of thepresent techniques. Accordingly, it is the following claims includingany amendments thereto that define the scope of the present techniques.

What is claimed is:
 1. A system for enabling a dual fisheye model andcalibration, comprising: a calibration module that is at least partiallyhardware, to simultaneously generate a first set of coefficients of aninverse distortion polynomial representing radial distortion and asecond set of coefficients of an alternative distortion polynomial toproject a point from a fisheye image to a three-dimensional space. 2.The system of claim 1, wherein the first set of coefficients and thesecond set of coefficients are optimized via an iterative non-linearoptimization.
 3. The system of claim 1, wherein the first set ofcoefficients and the second set of coefficients are generated via adirect estimation.
 4. The system of claim 1, wherein the calibrationmodule enables offline calibration with a calibration chart or anotherdevice.
 5. The system of claim 1, wherein the calibration module enablesa dynamic calibration via an image captured by a camera of a cameraarray.
 6. The system of claim 1, where the inverse distortion polynomialincludes even powers, and excludes odd powers.
 7. The system of claim 1,further including a camera sensor to capture the fisheye image, whereinan image capture model of the camera sensor includes a projectivetransform from an ideal image plane to a two-dimensional (2D) image. 8.The system of claim 1, wherein an initial estimate of the first set ofcoefficients and the second set of coefficients is calculated, and theinverse distortion polynomial and the alternative distortion polynomialare calibrated simultaneously using the initial estimate.
 9. The systemof claim 1, wherein a bundle adjustment is used to simultaneouslyoptimize the first set of coefficients and the second set ofcoefficients.
 10. The system of claim 1, wherein a bundle adjustmentincludes minimizing a cost function to iteratively optimize the firstset of coefficients and the second set of coefficients.
 11. A method forenabling a dual fisheye model and calibrator, comprising: calibratingcoefficients of an inverse distortion polynomial, coefficients of analternative distortion polynomial to project a point from a fisheyeimage to a three-dimensional space, and camera parameters simultaneouslyto model a real-world distortion of at least one fisheye camera.
 12. Themethod of claim 11, wherein the camera parameters include sensorgeometry parameters and alignment parameters.
 13. The method of claim11, wherein the coefficients of the inverse distortion polynomial, thecoefficients of the alternative distortion polynomial, and the cameraparameters are generated via iterative non-linear optimization.
 14. Themethod of claim 11, wherein the coefficients of the inverse distortionpolynomial, the coefficients of the alternative distortion polynomial,and the camera parameters are generated via a direct estimation.
 15. Themethod of claim 11, wherein simultaneously calibrating the coefficientsof the inverse distortion polynomial, the coefficients of thealternative distortion polynomial, and the camera parameters to model areal-world distortion of the at least one fisheye camera is performedvia an offline calibration with a calibration chart or another device.16. The method of claim 11, wherein simultaneously calibrating thecoefficients of the inverse distortion polynomial, the coefficients ofthe alternative distortion polynomial, and the camera parameters tomodel a real-world distortion of the at least one fisheye camera isperformed via a dynamic calibration.
 17. The method of claim 11, whereinthe inverse distortion polynomial includes even powers, and excludes oddpowers.
 18. The method of claim 11, wherein an image capture model of acamera sensor of the at least one fisheye camera includes a projectivetransform from an ideal image plane to a 2D image.
 19. The method ofclaim 11, wherein a bundle adjustment is used to simultaneously optimizethe coefficients of the inverse distortion polynomial and thecoefficients of the alternative distortion polynomial.
 20. The method ofclaim 11, wherein bundle adjustment including minimizing a cost functionis used to simultaneously optimize the coefficients of the inversedistortion polynomial and the coefficients of the alternative distortionpolynomial.
 21. An apparatus, comprising: a camera array including atleast one fisheye camera; and a processor to simultaneously generate afirst set of coefficients of an inverse distortion polynomialrepresenting a radial distortion of the at least one fisheye camera anda second set of coefficients of an alternative distortion polynomialrepresenting the at least one fisheye camera to project a point from afisheye image to a three-dimensional space.
 22. The apparatus of claim21, wherein the first set of coefficients and the second set ofcoefficients are optimized via an iterative non-linear optimization. 23.The apparatus of claim 21, wherein the first set of coefficients and thesecond set of coefficients are generated via a direct estimation. 24.The apparatus of claim 21, wherein the processor enables offlinecalibration of the camera array with a calibration chart or anotherdevice.
 25. The apparatus of claim 21, wherein the processor enables adynamic calibration via an image captured by the at least one fisheyecamera of the camera array.